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Background 

The first paper will provide a general summary of the different methods for identifying relevant 
studies for evidence reviews and the various options for making decisions about the 
characteristics of included studies. Comparisons of the search strategies employed by several 
research clearinghouses will be provided, highlighting the nature of evidence that results from 
the different search strategies. The presenters will use studies of interventions in postsecondary 
education to highlight how different conclusions might result from employing different methods 
of identifying studies. 

One of the primary objectives of the various research clearinghouses, including the What Works 
Clearinghouse, is to provide evidence to the public suitable for use in decision-making. As the 
use of evidence-based repositories for decision-making becomes more commonplace, the 
organizations may also become de facto gatekeepers of infonnation. That is, decision-makers 
may only have access to what the clearinghouses decide to put forward as evidence. Considering 
the implications the gatekeeper role of clearinghouses in the context of how evidence might be 
used by the public is the topic of this paper. 

Purpose 

The purpose of this investigation is to report about study identification practices across evidence- 
based registries and clearinghouses in social policy fields, which serve as a resource for 
scientific, evidence-based decision-making about practices about desired outcomes in these 
social policy fields. The information retrieval procedures of the What Works Clearinghouse will 
be described in detail. Using the general study identification and search procedures for the WWC 
and other evidence-based repositories, the presenters will highlight how differences in 
comprehensiveness and transparency can result in different studies available for review and 
synthesis. 

Population 

Study identification practices from eight evidence registries will be analyzed and reported. The 
registries to be included are: (1) Blueprints for Healthy Youth Development, (2) the California 
Evidence-Based Clearinghouse (CEBC), (3) the Coalition for Evidence-Based Policy, (4) 
CrimeSolutions.Gov, (5) National Registry of Evidence-Based Programs and Practices (NREPP), 
(6) the Promising Practices Network (PPN), (7) What Works in ReEntry, and (8) What Works 
Clearinghouse (WWC). Table 1 provides a summary of the general study identification practices 
for the evidence-based repositories included in this investigation. 
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Table 1 



Registries 

Type of Study 
Identification Process 
(Literature review 
and/or Nomination) 

Search by Programs or by 
Studies (in particular 
outcome/topic domains) 

1 . Blueprints for Healthy 
Youth Development 

Literature review and 
nominations 

Studies by outcomes of interest 

2. California Evidence- 
Based Clearinghouse 

Literature review 

Programs and interventions by 
topic areas 

3. Coalition for Evidence- 
Based Policy 

Nominations 

Studies on targeted social policy 
issues 

4. CrimeSolutions.Gov 

Literature review and 
nominations 

Programs and practices by fields 
of interest 

5. National Registry of 
Evidence-Based 
Programs and Practices 

Nominations 

Programs and interventions by 
fields of interest 

6. Promising Practices 
Network 

Nominations 

Programs by outcomes of 
interest 

7. What Works in ReEntry 

Literature review 

Studies by topic areas in the 
clearinghouse framework 

8. What Works 
Clearinghouse 

Literature review 

Studies by established topics 


Research Design, Data Collection, and Analysis 

Information about study identification processes across eight registries will be compiled through 
analysis of websites, and follow-up contact with contact persons for various registries through 
email or phone inquiry. Website documents and follow-up information will be used to uncover 
the specific strategies used for electronic database searches, identification of unpublished studies, 
and how the different repositories handle nominations from researchers, program developers, and 
the public. 

For registries employing electronic database searches, the following research questions will be 
addressed: 

1) What databases are used to conduct searches? (e.g., PsycINFO, ERIC, etc.) 

2) How often is the literature scanned for each topic area or field of interest? 

3) How are search terms developed and revised? 

4) According to what criteria are study abstracts screened? 

5) According to what criteria are studies rated as eligible for review? 

6) What is the date range of electronic database searches? (e.g., 1994 for a twenty year time 
frame) 


2 Funding for the Promising Practices Network ended in June 2014; however, information is archived on the website. 
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For registries employing nominations, or some combination of nomination and electronic 
database searches, questions to be addressed include: 


1) After a program or intervention has been nominated, are additional literature scans 
conducted to identify additional studies? 

2) If other studies are located about an intervention or program, how are these considered? 

In addition, because publication bias can have significant impacts on the impact estimates 
reported in a systematic review, the presentation will discuss how (or if) unpublished research is 
included in the search strategies employed by the different repositories and the methods for 
collecting such research. In the postsecondary education field, unpublished research is common. 
Using postsecondary education research reviewed by the What Works Clearinghouse, the 
presenters will demonstrate What Works Clearinghouse search procedures and highlight benefits 
of a comprehensive and systematic search. 

Table 2 lists eight evidence-based practice repositories, their general policies on the inclusion of 
unpublished research, and the number of studies included in the repository. 


Table 2 



Registries 

Publication 

Requirements 

Number of Studies 
Reviewed/Results of Searches 
(per website on 09/28/14) 

1 . Blueprints for Healthy 
Youth Development 

Does not specify 
publication requirement 

More than 1,250 programs 
assessed since founding in 1996 

2. California Evidence- 
Based Clearinghouse 

Published, peer-reviewed 
studies only. 

339 programs have been rated 

3. Coalition for Evidence- 
Based Policy 

Published or unpublished 

2 1 interventions have been rated 
in “top tier” or “near top tier” 

4. CrimeSolutions.Gov 

Studies published in 
English in a peer- 
reviewed journal or other 
professional publication, 
or comprehensive 
evaluation report 

319 programs have been rated; 
and 26 practices have been rated 

5. National Registry of 
Evidence-Based 
Programs and Practices 

Studies published in 
English in a peer- 
reviewed journal or other 
professional publication, 
or comprehensive 
evaluation report 

339 interventions have been 
rated 

6. Promising Practices 
Network 3 

Studies are publicly 
available 

28 proven programs 

7. What Works in ReEntry 

Studies conducted by 
independent researcher or 

55 interventions and 50 
evaluations. 


3 Since the Promising Practices Network is no longer in operation, follow-up questions may be limited to information that may be 
currently provided by RAND staff who fonnerly worked on the project, or available only on the archived website. 


SREE Spring 2015 Conference Abstract Template 


3 





published in a peer- 
reviewed journal 


8. What Works 
Clearinghouse 

Studies published in 
English in a peer- 
reviewed journal or other 
professional publication, 
or comprehensive 
evaluation report 

552 interventions and 10,310 
studies since 2005 


Results and Conclusions 

Among these eight registries, four use literature reviews, three use nominations, and one uses a 
combination of literature reviews and nominations for study identification. Four registries 
conduct searches for studies by topic area or outcome; and four registries conduct searches by 
programs or interventions, and then apply study appraisal guidelines. 

Three registries specify that publications must be in English in a peer-reviewed journal or other 
professional publication, or comprehensive evaluation report (Crimesolutions.gov, National 
Registry of Evidence-Based Programs and Practices, and What Works Clearinghouse). One 
registry specifies studies must be in peer-reviewed journals (California Evidence-Based 
Clearinghouse). Unpublished studies may be considered by Blueprints, the Coalition for 
Evidence-Based Policy, and the Promising Practices Network (if publicly available). 
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